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Overview 



The Bay Area School Reform Collaborative (BASRC, now called Springboard Schools) in San 
Francisco, California, is a grant-making organization that supports districtwide efforts to improve 
the quality and equity of student outcomes. The organization pursues various reform strategies. 
This report discusses the “focal strategy,” which targeted selected “focal districts” in the Bay 
Area, beginning in the 2002-2003 school year, and was designed to increase the intensity of ear- 
lier BASRC reforms by creating more opportunities for district and school administrators to 
interact with BASRC staff. The focal strategy does not prescribe a particular curriculum or school 
structure. Instead, it promotes a vision of culture change, relying on three key features: coaching 
of district and school leaders; evidence-based decision-making at all levels of the system; and 
networking within and across schools to share experiences and lessons. 

With funding from the William and Flora Hewlett Foundation, MDRC, a nonpartisan, nonprofit 
education and social policy research organization, conducted an independent evaluation of 
BASRC’ s focal strategy. This report, the second of two, analyzes the relationship between the 
focal strategy and improvements in student reading achievement. It compares progress in the focal 
districts during the three years of the strategy’s implementation to progress in a set of carefully 
chosen comparison districts in the same area over the same period. Though differences in the out- 
comes cannot necessarily be attributed to the BASRC focal strategy, the comparison illuminates 
the relationships between student outcomes and the focal strategy. The key findings are: 

• In the districts that participated in the focal strategy, there were improvements in reading 
achievement during the years of the focal strategy, but these improvements were either 
similar to or only slightly greater than improvements in similar districts in the Bay Area. 

• The evident lack of a substantial, pervasive association between the BASRC focal strategy 
and student achievement may not be surprising given that the strategy primarily targeted 
district leadership, was not sustained at the school level, and did not specify particular in- 
structional practices or supports at the school or classroom level. 

• Overall, the implementation research suggests that, in practice, the intensity of the interven- 
tion, the consistency of focus on improving teaching and learning, and the connection be- 
tween the district-level focal reforms and changes in daily school life were not sufficiently 
realized. 

BASRC faced a variety of challenges in translating district-level coaching, evidenced-based 
decision-making, and networking and collaboration into changes in teaching and learning. Find- 
ings from this report indicate that the BASRC focal strategy had limited capacity to improve 
student performance and close achievement gaps beyond trends that were already in motion. 
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Preface 



More and more school districts are developing reform strategies to enhance leadership 
at all levels, improve student achievement, and close achievement gaps. Many use the services 
of external support organizations like the Bay Area School Reform Collaborative (BASRC, 
now known as Springboard Schools) to create plans for making change. BASRC’ s “focal strat- 
egy,” the subject of this report, is a reform effort that was implemented in several districts 
throughout the San Francisco Bay Area, beginning in 2002. Emphasizing process rather than 
particular curricula, teaching strategies, or accountability systems, the focal strategy relied on 
three key features: coaching of district and school leaders, using evidence to inform decisions 
about strategies for improving achievement. 

But is the BASRC focal strategy associated with districtwide improvements in average 
elementary student achievement? This report builds on earlier findings, released in February 
2006, which found no strong association between the focal strategy and changes in average stu- 
dent proficiency rates on state-mandated literacy tests. The findings presented in this report, 
with an additional year of follow-up, point to the same conclusion: In the districts that partici- 
pated in the focal strategy, there were improvements in achievement during the years of the fo- 
cal strategy, but these improvements were either similar to or only slightly greater than those in 
similar districts in the Bay Area. 

Nevertheless, MDRC’s evaluation of the BASRC focal strategy expands knowledge 
about how districts approach systemic reform. In addition, it sheds light on how difficult it is for 
external support organizations to partner with districts to help initiate and sustain systemic re- 
form. What difficulties do external support organizations like BASRC face in their efforts to 
forge effective coaching relationships, influence norms and cultures, introduce new processes, 
create coherence, and build knowledge? How do the organizations meet those challenges? This 
report speaks to those questions, outlining the challenge of translating systemic changes into 
reforms that reach the classroom, improve instruction, and raise student achievement. 

If system-level changes are to result in improvements in student achievement, this study 
suggests that external support organizations like BASRC might be more successful if they sup- 
port both district offices and schools and help them identify instructional strategies that reach 
the classroom. 



Gordon L. Berlin 
President 




Acknowledgments 



The authors of this report would like to thank the Bay Area districts that are the focus of 
this study: Alameda Unified School District, Newark Unified School District, Pacifica School 
District, San Bruno Park Elementary School District, and San Rafael City Elementary School 
District, as well as the schools within these districts. We are grateful to the administrators, 
coaches, and teachers in these districts for sharing their time, experiences, and perceptions. 

We also thank the William and Flora Hewlett Foundation for its financial support of the 
evaluation of the Bay Area School Reform Collaborative (BASRC). Special acknowledgment is 
due to Marshall Smith, Ray Cortinas, Phoenix Wang, Jorge Ruiz-de- Velasco, and Kristi Kim- 
ball for their insights and for their valuable comments throughout the evaluation. 

Many BASRC staff, including Merrill Vargo, Ida Oberman, Rick Demelio, Carlene 
Gundersgard, Brian Simmons, Katherine Barr, and Marie Kawase, were very generous with the 
time they spent clarifying components of their reform efforts and their ongoing implementation 
work. We are also grateful for their comments on the preliminary findings. 

We would also like to thank Joan Talbert, Milbrey McLaughlin, Pai-rou Chen, Wendy 
Lin, and the rest of the research team at Stanford University’s Center for Research on the Con- 
text of Teaching for their collaboration in this study. Their expertise in survey design, interview 
skills, and extensive knowledge of districts and schools greatly informed our study. We also 
appreciate the access they allowed us to their rich field notes, as well as their comments on re- 
port drafts. 

Brian Uslan at the California Department of Education was instrumental in acquiring 
the public data files used in our analysis. 

Finally, at MDRC, Christian Geckeler, Derek Chan, Kathryn Ferreira, and Vivian 
Mateo coordinated production of this report, prepared tables and figures, and helped execute 
data analyses. Andrea Dymess conducted interviews and focus groups in the school districts 
and schools. James Kemple, Robert Ivry, Fred Doolittle, John Hutchins, Margaret Bald, Janet 
Quint, and Howard Bloom reviewed analysis plans and early drafts of the report. We are grate- 
ful to them for their helpful guidance on the content, methodology, and organization of the re- 
port. In addition, we would like to thank Amy Rosenberg for her careful editing and Stephanie 
Cowell for helping to prepare the final text for publication. 



The Authors 




Executive Summary 



This is the second and final report for MDRC’s evaluation of the Bay Area School Re- 
form Collaborative (BASRC), a grant-making and support organization in San Francisco, Cali- 
fornia. 1 BASRC is dedicated to improving student achievement in public schools and narrowing 
achievement gaps among different racial, ethnic, and socioeconomic groups. BASRC reforms 
are designed as continuous improvement processes driven by collective inquiry into student 
learning outcomes, school functioning, and teacher practice. This process-oriented reform strat- 
egy is designed to build professional knowledge about effective practice, promote mutual ac- 
countability, and facilitate collaboration. BASRC hypothesizes that making changes along these 
lines will result in improved teaching and learning, but the organization does not prescribe par- 
ticular instructional practices, curricula, or school structures that should evolve as a result of its 
reform processes. 

This evaluation centers on a specific BASRC initiative, the “focal strategy,” as it was 
implemented during the 2002-2003, 2003-2004, and 2004-2005 school years in five Bay Area 
districts (the “focal districts”), 2 and two to four selected schools within those districts (the “focal 
schools”). 3 Building on earlier phases of BASRC reforms, which began in 1996, the focal strat- 
egy was meant to increase the intensity of the core BASRC reforms by creating more opportu- 
nities for district and school administrators to interact with BASRC staff. In selecting the focal 
districts, BASRC tended to focus on districts where a strong working relationship had devel- 
oped during earlier BASRC efforts, and the focal reforms were implemented on top of existing 
BASRC reform work in these places. In short, compared with the earlier reform strategies, the 
focal strategy served as a stronger test of BASRC reform ideas in fewer places. 

While the first report on the evaluation explored the relationship between the BASRC 
focal strategy and student reading achievement for third- and fifth-graders in the first two years 
of implementation in the focal districts, this report focuses only on fifth-graders and provides an 
additional year of follow-up data. 

The key findings of MDRC’s evaluation of the BASRC focal strategy are as follows: 



'During this study, BASRC changed its name to Springboard Schools and became a nonprofit, fee-for- 
service organization that offers its clients — urban, suburban, and rural school districts throughout California 
— professional development services, on-site district coaching, access to research, and a variety of tools and 
services for improving schools. For ease of reference, the report uses the former name throughout. 

2 While the BASRC focal strategy continued during the 2004-2005 school year, it began to wind down 
during that period, as BASRC focused on developing new professional development services and strategies for 
a broader set of districts in California. See www.springboardschools.org. 

3 During the 2003-2004 school year, BASRC added a sixth focal district that is not included in MDRC’s study. 
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• In the districts that participated in the focal strategy, there were improvements 
in achievement during the years of the initiative, but these improvements were 
either similar to or only slightly greater than improvements in similar districts 
in the Bay Area that were not part of the focal strategy. 

• The evidence suggests that the BASRC focal strategy is not associated with 
substantial districtwide improvements in average elementary student achieve- 
ment on state standards tests of reading. 

• Although BASRC originally intended to support both districts and schools in a 
“bottom-up and top-down” approach, in practice the focal strategy primarily 
served district leadership, was not sustained at the school level, and, thus, typi- 
cally did not lead to specific changes in instruction or specific instructional 
supports at the school and classroom levels. 

MDRC’s evaluation of the BASRC focal strategy expands knowledge about how dis- 
tricts approach systemic reform. BASRC’ s focal strategy is an example of how external support 
organizations are increasingly partnering with districts to help initiate and sustain systemic re- 
form. This report provides new knowledge about the difficulties external support organizations 
like BASRC face in their efforts to forge effective coaching relationships, influence norms and 
cultures, introduce new processes, create coherence, and build knowledge. Moreover, the report 
demonstrates the challenge of translating systemic changes into reforms that actually reach the 
classroom, improve instruction, and raise student achievement. 



The BASRC Focal Strategy 

The BASRC focal strategy centered on three core features: 

• coaching of superintendents, district and school leaders, and teachers 

• evidence-based decision-making at all levels of the system (for instance, us- 
ing student achievement data to inform decisions about policy and practice) 

• networks and collaboration among administrators and teachers and within 
and across districts and schools 

All three features were also part of BASRC’ s earlier reform efforts. Once the focal 
strategy began, however, school-level coaching by BASRC staff was redirected toward focal 
schools within the focal districts. In addition, “executive coaches” (former superintendents) 
were hired to advise and support the leaders in the focal districts. Along with regular meetings 
on comprehensive needs assessments and reviews of progress, these efforts were intended to 
reinforce all elements of the strategy in the focal districts and schools. 
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The theory of action underlying the BASRC focal reform strategy posits that the im- 
plementation of these three design features yields changes in district, school, and classroom 
practices that in turn improve student outcomes. When instituted together, coaching, evidence- 
based decision-making, and networks and collaboration are thought to increase districts’ and 
schools’ engagement in a continuous improvement process that is directed at improving the de- 
gree and equity of student achievement levels. Because the strategy is process-oriented, it can 
result in different policies or practices that may or may not be linked with specific approaches to 
teaching and learning. 



The Evaluation of the BASRC Focal Strategy 

The independent evaluation of BASRC is funded by the William and Flora Hewlett 
Foundation and includes studies by both MDRC and the Stanford University Center for Re- 
search on the Context of Teaching (CRC ). 4 The CRC studies focus on the process of reform as 
it relates to BASRC in general and the relationship between various BASRC reform efforts and 
changes at the district, school, and classroom levels. MDRC’s study attempts to shed light on 
the relationship between the BASRC focal strategy and improvements in student achievement. 

It is important to note that MDRC’s study is limited in its capacity to establish a rigor- 
ous causal relationship between implementation of BASRC focal reforms and changes in stu- 
dent achievement that may have occurred after the focal strategy was introduced. BASRC se- 
lected districts where it already had a strong working relationship and where the superintendent 
was reform-minded and focused on achievement gaps. Therefore, the BASRC focal districts 
may have differed from comparison districts with similar demographic and achievement charac- 
teristics in unmeasured and meaningful ways. Consequently, subsequent differences between 
the progress made in the focal districts and progress made in comparison districts could be due 
to the fact that the BASRC focal districts had particularly entrepreneurial leaders rather than to 
additional contributions made by the BASRC focal strategy. Thus, the analyses undertaken for 
this evaluation are intended to be suggestive and descriptive of the potential relationship (or lack 
of relationship) between implementation of the BASRC focal strategy and trends in student 
achievement, rather than definitive estimates of the causal effect of the BASRC focal strategy 
on student achievement. 

The BASRC focal reforms tended to center on literacy instruction and were concentrated 
at the elementary level. Therefore, all analyses of the focal strategy are confined to the elementary 
school level and emphasize district performance on measures of student literacy. The analyses of 
student achievement examine outcomes particularly for fifth-grade students, who have been in the 



4 CRC also conducted an evaluation of BASRC’ s first phase of reform work ( 1995-2001). 
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schools longer than other students at the elementary school level and whose scores reflect a more 
cumulative effect of the quality of education across the elementary grades. 



Key Findings 

Implementation Findings 

MDRC’s implementation findings are based primarily on interviews and focus groups 
conducted in four of the five focal districts during the 2003-2004 and 2004-2005 school years 
— the second and third years of the BASRC focal strategy. 

• At the district level, the BASRC focal strategy resulted in coaching of super- 
intendents and other district administrators, professional networking oppor- 
tunities for district and school leaders, and needs assessment and review 
meetings with BASRC staff. However, the strategy did not typically translate 
into specific reforms aimed at improving teaching and learning. 

Executive coaches were charged with helping superintendents identify, implement, and 
support system-wide efforts to raise student achievement and narrow achievement gaps. In prac- 
tice, the work between the executive coaches and superintendents tended to center on building 
capacity for these efforts — such as strengthening leadership, becoming more focused on re- 
form work, and improving communication — rather than on actually implementing reforms. 
The executive coaches, along with BASRC staff coaches, also began meeting with other district 
leaders who had a more direct role in affecting teaching and learning. This coaching often re- 
volved around a variety of district needs related to readiness for education reforms, such as 
building support for district reform goals and mending communication difficulties within and 
across departments. In interviews, executive coaches said they made progress in these areas and 
putting instructional reforms in place was “the next step.” All together, the coaching, network 
meetings, and meetings with BASRC staff led to the establishment of goals for improving stu- 
dent achievement and reducing achievement gaps and to enhanced capacity for working toward 
the goals. However, they did not typically translate into specific reforms aimed at improving 
teaching and learning. 

• By the 2004-2005 school year (the third year of the initiative), the focal dis- 
trict strategy had ended early, either formally or informally, in three of the 
five focal districts. 

In three of the five original focal districts, BASRC staff encountered difficulties that led to 
an early conclusion of the BASRC focal reform efforts. According to BASRC staff, they never 
fully gained traction in one of the focal districts and officially ended the partnership before the 
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start of the t h ir d year of the strategy. In another focal district, although the executive coach main- 
tained a mentoring role with the superintendent, the coaching role as BASRC envisioned it ended 
in September 2004. And, in a third focal district, staff turnover, both at the district office and 
among the BASRC staff working with the particular district, led to a disruption in the work. 

• By the 2003-2004 school years (the second year of the initiative), the school- 
level aspect of the focal strategy faded. Thus, the model evolved to be almost 
entirely a district strategy. 

In theory, coaching by BASRC staff was to be a primary feature of the school-level focal 
strategy. This “external” coaching effort was distinct from the other “internal” coaching efforts in 
place in non-focal schools (including school-level literacy coaches and coaches employed by the 
district to support reform work). However, in the first year of implementation, BASRC encoun- 
tered several complications, including resistance to the BASRC coaches from school-level staff. 
By the end of the first year, BASRC coaches did not have a significant coaching role in most of 
the focal schools. In the third year of the strategy, school-level coaches began to develop more 
prominent roles at elementary schools in two of the focal districts, but this coaching was not gen- 
erally focused on delivery, support, or implementation of the BASRC focal reforms. 

• At the schools within the focal districts, there was evidence of all three key 
features of the BASRC focal strategy; however, the instances of the key fea- 
tures were likely vestiges of earlier BASRC reform phases. 

MDRC found evidence that all three of the key features of the BASRC focal reform strat- 
egy — coaching by district or school staff, evidence-based decision-making, and networks and 
collaboration — were in place at the schools in the focal districts. However, field research data 
indicate that these were typically implemented as a result of participation in earlier BASRC re- 
form efforts rather than participation in the BASRC focal strategy itself. It is also possible that 
these reform practices were in place before any participation in BASRC reform efforts. Moreover, 
in the schools visited by MDRC, it was difficult to detect meaningful differences in the types of 
BASRC supports and reform activities in place in focal schools compared with non-focal schools. 

• The extent to which teachers engaged in BASRC practices — coaching by 
district or school staff, evidence-based decision-making, and networks and col- 
laboration — did not increase during the years of the focal strategy. 

In investigating implementation of the focal strategy at the school level, the evaluation 
also included an annual teacher survey developed by MDRC and CRC. The survey was admin- 
istered in BASRC elementary schools each spring in the same four focal districts in which field 
research was conducted. Among other things, the survey posed a variety of questions regarding 
the extent, nature, and frequency of teacher participation in school reform practices related to 
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BASRC’s three key features. The survey does not capture all activities related to the focal strat- 
egy, but it does provide information on some key activities. Overall, the survey data revealed 
little change in the key reform activities during the years of the focal strategy. 

Student Achievement Outcomes 

In order to explore the relationship between participation in the BASRC focal strategy 
and changes in student achievement, MDRC’s analysis of student achievement compared pro- 
gress in the BASRC focal districts in the three years of the BASRC focal reforms to progress in a 
set of carefully chosen comparison districts from throughout the San Francisco Bay Area over the 
same period. 

• Fifth-grade students’ achievement in reading improved over time in the focal 
districts, in terms of both an increase in the percentage of students scoring pro- 
ficient or above and a decrease in the percentage of students scoring below the 
basic level. There were similar improvements in the comparison districts. 

Both the focal districts and the Bay Area comparison districts saw improvements in 
fifth-grade language arts achievement at both the top and bottom of the achievement scale. In 
terms of proficiency rates, the average percentage of focal district students scoring at or above 
proficient increased from 38 percent to 54 percent by the end of follow-up Year 3. A similar 
change, from 39 percent to 52 percent, occurred in the comparison districts. (These patterns are 
in line with upward trends across the Bay Area and across the state as well.) Both the BASRC 
focal districts and the comparison districts reduced the number of students performing below 
basic. In Years 2 and 3, the improvements in the focal districts may have outpaced those that 
occurred in the comparison districts. For example, the percentage of students scoring below ba- 
sic dropped by about 5 percentage points in the focal districts compared with a drop of about 2 
percentage points in the comparison districts, suggesting a net difference of 3.2 percentage 
points. However, the difference between progress in the focal and comparison districts is rela- 
tively small and not statistically significant. As such, the possibility that the difference is simply 
the result of chance cannot be ruled out. In short, it is not possible to conclude that there is actu- 
ally a systematic association between implementation of the BASRC focal strategy and student 
achievement on California’ s state standards. Moreover, any association that does exist appears 
to be very modest in size. 

• Among blacks and Hispanics, English Language Learners (ELL), and eco- 
nomically disadvantaged fifth-grade students, the improvements in reading 
achievement in the BASRC focal districts outpaced the improvements in the 
comparison districts later in the follow-up period. However, the differences 
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in the improvements in the focal districts and those in the comparison dis- 
tricts are small and generally not statistically significant. 

When focusing on fifth-grade achievement among economically disadvantaged stu- 
dents, black and Hispanic students, and English Language Learners, performance in the focal 
districts appeared to surpass the improvements in the comparison districts in the second and 
third years of follow-up. The differences MDRC observed could suggest a positive association 
between the BASRC focal strategy and improvements in reading scores on the California Stan- 
dards Tests (CSTs) among disadvantaged subgroups of students. However, the size of the dif- 
ferences is small and the differences are generally not statistically significant, meaning that the 
differences could have occurred by chance. 

The findings are not surprising given that implementation of the strategy did not trans- 
late into instructional reforms or increased use of BASRC-related practices in the schools. 
Moreover, even if the small difference between the focal and non-focal districts points to an 
association, MDRC cannot make causal inferences. The selection of “reform-minded” districts 
as participants in the focal strategy could mean that the BASRC focal districts were more likely 
than districts with similar student populations and achievement track records to implement re- 
forms and improve student performance. 

• Schools with higher levels of BASRC-related practices did not have higher 
levels of achievement. 

Using survey measures of reform practices related to BASRC’ s three key features, 
MDRC examined the relationship between schools’ average levels of BASRC-related practices 
during Years 1, 2, and 3 of the focal strategy and proficiency levels in reading in Year 3, con- 
trolling for differences in the demographic characteristics of the students these schools served 
and for achievement levels prior to the start of the focal strategy. This analysis did not reveal 
relationships between the BASRC-related practices measured on the survey and fifth-grade 
reading achievement. The absence of such relationships may have several explanations. Lirst, 
the measures may not have sufficiently captured enough variation across schools. Lor example, 
all schools had high average ratings for their tendency to examine achievement data, which may 
or may not be because they were all BASRC grantees. Therefore, the analysis assesses only 
whether improving from a high level to a higher level of examining achievement data is related 
to better achievement outcomes; it does not assess whether improving from a low level to a high 
level of examining achievement data is related to better achievement. The absence of a relation- 
ship could also reflect limitations of the survey measures to sufficiently capture teachers’ prac- 
tices. However, it could also indicate that the BASRC-related practices are not associated with 
improved proficiency in reading. 
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Interpreting the Findings 

This report builds on an earlier report, released in February 2006, which found no 
strong association between the BASRC focal strategy and changes in average student profi- 
ciency rates on state -mandated literacy tests. In this report, with an additional year of follow-up, 
the evaluation points to the same conclusion. In the districts that participated in the focal strat- 
egy, there were improvements in achievement during the years of the focal strategy, but these 
improvements were either similar to or only slightly greater than improvements in similar dis- 
tricts in the Bay Area. 

Given the nature of the reforms and the implementation patterns, the findings may be un- 
derstandable. By design, the model did not focus on implementing particular classroom instruc- 
tional strategies; rather, it emphasized a process through which districts, schools, and teachers 
could come to learn how they can best support improved teaching and learning. In practice, the 
BASRC focal strategy primarily targeted district leadership and focused primarily on building 
capacity for reform efforts rather than on actually implementing reforms directed at teaching 
and learning. Also, at the school-level, coaching by BASRC staff fell by the wayside, and, al- 
though a goal of the focal strategy was to intensify the core BASRC reforms, it was unclear 
whether the BASRC-related practices at the school level resulted from the focal strategy. In addi- 
tion, the teacher survey data indicated little change in BASRC-related practices during the years of 
the focal strategy. 

Systemic reforms such as BASRC can take a long time to take root. If they do, and if 
they are translated into effective instructional improvements, the changes in teaching and learn- 
ing could be substantial and more sustainable than those evoked by other reforms. This evalua- 
tion suggests that external support organizations like BASRC might be more successful in help- 
ing districts achieve successful systemic reforms if they can simultaneously support both district 
offices and schools, if they can move beyond building capacity, and if they can help district of- 
fices and schools identify and implement effective instructional changes. 
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Chapter 1 

Introduction 



This is the second and final report for MDRC’s evaluation of the Bay Area School Re- 
form Collaborative (BASRC), a grant-making and support organization in San Francisco, Cali- 
fornia. 1 Although BASRC has launched several reform efforts since its inception, this evalua- 
tion is focused on a particular initiative, the “focal strategy,” which was implemented in Bay 
Area districts starting in the 2002-2003 school year. The focal strategy is a reform effort aimed 
at building capacity across entire school districts and at all levels of the system for a systematic 
and continuous improvement process. The strategy also concentrates BASRC staff resources in 
a small number of districts (the “focal districts”), and in schools within those districts (the “focal 
schools”), in order to provide intensive and responsive coaching support to teachers. 

This report builds on an earlier report, released in February 2006, which found no 
strong association between the BASRC focal strategy and districtwide improvements in average 
elementary student reading achievement. In this report, with an additional year of follow-up, 
MDRC reaches the same conclusion. While the first report explored the relationship between 
the BASRC focal strategy and student achievement for third- and fifth-graders in the focal dis- 
tricts, this report considers only fifth-graders in the focal districts. 

In the districts that participated in the focal strategy, there were improvements in 
achievement during the years of the focal strategy, but these improvements were either similar 
to or only slightly greater than improvements in similar districts in the Bay Area. Given the na- 
ture of the BASRC focal reforms, the lack of a strong, pervasive relationship with student 
achievement may be understandable. In practice, the strategy primarily targets district leader- 
ship and does not specify how reform activities may lead to specific changes in instruction or 
specific instructional supports. Moreover, the research suggests that the intensity of the inter- 
vention, the consistency of focus on improving teaching and learning, and the connection be- 
tween the district-level focal reforms and changes in daily school life were not sufficiently real- 
ized. While some key BASRC reform practices were prevalent in schools within the focal dis- 
tricts, they were likely vestiges of earlier BASRC reform phases. In this report, MDRC’s study 
attempts to assess the relationship among some of these core BASRC reforms, but it does not 
evaluate the effect of earlier BASRC reform phases. Rather, the study concludes that the focal 
strategy, which was designed to increase the intensity of the core BASRC reforms by creating 



'While the study was being conducted, BASRC changed its name to Springboard Schools. For ease of ref- 
erence, the report uses the former name throughout. 
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more opportunities for district and school administrators to interact with BASRC staff, is not 
associated with changes in student achievement. 



Overview of BASRC Reforms 

BASRC is dedicated to improving student achievement in public schools and narrowing 
achievement gaps among different racial, ethnic, and socioeconomic groups. BASRC promotes 
a vision of culture change in which teachers and administrators engage in a collaborative and 
ongoing reform process at every level of the education system. BASRC’ s reforms do not pre- 
scribe predetermined instructional practices, curricula, or school structures as a “treatment.” 
Rather, they are designed as continuous improvement processes driven by collective inquiry 
into student learning outcomes, school functioning, and teaching practices. According to 
BASRC, its process-oriented reform strategy builds professional knowledge of effective prac- 
tice, mutual accountability and collaboration, and continuous improvement of the quality and 
equity of student outcomes. Importantly, while BASRC hypothesizes that these changes in cul- 
ture will translate into improved student learning, the organization does not specify the particu- 
lar mechanisms or pathways through which they translate into district changes in teaching. 

At the district level, BASRC focal reforms provide an example of a role that commu- 
nity organizations, external to school districts, may play in helping districts initiate and sustain 
improvement efforts. Such organizations, often referred to as “district support organizations” or 
“intermediaries,” typically provide a reform vision and assistance for bringing about systematic 
changes, but they can vary widely in their missions and approaches . 2 In the case of the focal 
strategy, BASRC’ s vision was a systematic and continuous improvement process for raising 
achievement and closing achievement gaps, and it provided assistance primarily in the form of 
coaching, as well as professional development for district and school leaders, networking oppor- 
tunities, and materials. In terms of both the vision and the approach, BASRC’ s focal strategy 
may be comparable with the efforts of the Institute for Leadership (IFL) and the Stupski Foun- 
dation. Both of these intermediary organizations also seek cultural changes and new processes 
focused on improving student achievement and closing achievement gaps, and both organiza- 
tions provide technical assistance to district leaders in the form of coaching or advising, as well 
as networking opportunities and materials . 3 



2 The term “district support organization” was introduced by Kronley and Handley (2003). Their term, as 
well as the term “intermediary,” typically refers to a wide range of “outside organizations — public, quasi- 
public, private for-profit, and private nonprofit — that seek to engage or are engaged by school districts and 
efforts at systematic reform” (Kronley and Handley, 2003, p. 4). 

3 Stupski Foundation Web site (2006); Marsh et al. (2005). 
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BASRC focal reforms are distinct from prescriptive classroom-level reforms, such as 
“Success for All,” which aim to implement particular classroom-level instructional reforms de- 
signed to create specific changes in classroom instructional practice in order to improve student 
achievement and reduce racial and economic achievement gaps. 4 Although BASRC reforms are 
implemented at both the school and the district levels, they are conceptually closer in approach 
to the initial idea of the “Accelerated Schools” model, in that that model is not necessarily built 
around a single definition of effective teaching practice but instead draws on external and inter- 
nal coaches to facilitate a process for school improvement that is intended to improve teaching. 5 

BASRC reforms have been implemented in Bay Area schools and districts in several 
phases over the past nine years (see Figure 1.1). Initially incorporated in early 1995, BASRC re- 
ceived $50 million in matched grants from the William and Flora Hewlett Foundation and the 
Annenberg Foundation. During Phase I (1996-2001), BASRC awarded grants to 87 schools over 
three to five years. At this stage, BASRC reforms focused on school-level change and centered on 
inquiry as the “engine” for improving school practice and student achievement. 6 In 2001, the or- 
ganization received an additional $40 million ($25 million and $15 million from the two founda- 
tions, respectively) to embark on its second phase of reform work in the Bay Area. In Phase II 
(2001-2005), BASRC began to focus more on the district as an agent for change and for scaling 
up reform. BASRC expanded its focus into a coordinated school- and district-level reform model 
in response to feedback from the Phase I evaluation that suggested schools could not sustain im- 
provement without active district support. 7 As part of Phase II, BASRC awarded grants to 18 dis- 
tricts to support 23 clusters of schools within those districts (a total of 91 schools) in working to- 
gether on reform efforts. These districts, selected from a pool of Bay Area grant applicants, re- 
ceived annual, renewable, multiyear improvement grants, which ranged from $125,000 to 
$500,000 depending on the number of BASRC schools in the BASRC district grant. 8 

While Phase II was under way, during the 2002-2003 school year, BASRC also em- 
barked on a focal strategy. This strategy directed BASRC staff resources to six “focal districts” 
and 19 “focal schools” within those districts. 9 While the focal districts and schools did not 
receive any additional BASRC funding, they did, with the introduction of the focal strategy, 



4 Slavin, Madden, Dolan, and Wasik (1993). 

5 Bloom (2002). 

6 Center for Research on the Context of Teaching (2002). 

7 Masten and Rendel (2002). 

BASRC also awarded grants of $2.1 million over three years to another nine school districts to support 
them in research and development efforts aimed at discovering how to change their own policies and practices 
to better support school improvement. 

9 Phase II schools may or may not have been funded in Phase I. Likewise, focal schools may or may not 
have been funded at the start of Phase II. MDRC’s study includes five of the six focal districts. 
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Figure 1.1 

Timeline of BASRC Reform Efforts 




receive increased staff coaching time. (BASRC decreased individualized support of its broader 
pool sample of BASRC grantees in order to reallocate more staff time to the focal sites.) Table 
1.1 summarizes the key dimensions of BASRC Phase I, Phase II, and the focal strategy. 

While the initiatives were in operation, BASRC maintained a consistent vision of rais- 
ing overall achievement and closing achievement gaps; however, the strategy for attaining this 
goal changed over time. BASRC’ s focus, or unit of change, shifted from individual schools to 
groups of schools (referred to as Local Collaboratives) to districts (and schools within those dis- 
tricts). The reforms developed over these phases, but remained centered on pursuing evidence- 
based decision-making (BASRC’ s Cycle of Inquiry) and building collaboration. During Phase I, 
BASRC encouraged schools to employ outside support providers but did not initially engage in 
direct technical assistance. After acknowledging that some schools needed assistance to achieve 
their goals, BASRC started offering school-level coaching toward the end of Phase I. 10 The 
coaching component became an integral part of BASRC’ s delivery strategy in Year 1 of Phase 
II and, in turn, in the focal strategy. In fact, BASRC developed the focal strategy to offer more 
support to districts and schools than had been offered in earlier phases of reform. In this way, 
BASRC concentrated and expanded many of the ideas developed for Phase II. The focal strat- 
egy was designed to increase the extent to which BASRC reform ideas would be carried out by 
increasing contact with BASRC staff, particularly at the district level. Thus, the focal strategy 
serves as a stronger test of BASRC reform ideas in fewer places. 



“’Center for Research on the Context of Teaching (2002). 
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Table 1.1 



Comparison of BASRC Reform Efforts 





Phase I 


Phase II 






Ongoing Phase II 


Focal Strategy 


Years 


(1996-1997 to 2000-2001) 


(2001-2002 to 2004-2005) 


(2002-2003 to 2004-2005) 


Vision 


Raise achievement and 
close achievement gaps 


Raise achievement and close 
achievement gaps 


Raise achievement and close 
achievement gaps 


Unit of Change 


Schools 


Local collaboratives (clusters 
of schools within district and 
district office) 


Districts 


Breadth 


87 schools 


23 local collaboratives (91 
schools in 18 districts) 


6 focal districts and 19 focal 
schools within them 


Key Dimensions of 
Reform 


Evidence-based decision- 
making 

Distributed leadership 
Professional community 


Coaching 

Evidence-based decision- 
making 

Networks and collaboration 


Coaching 

Evidence-based decision- 
making 

Networks and collaboration 


Delivery System 








BASRC Staff 
Coaching 


Direct assistance to leader- 
ship schools not achieving 
review-of-progress (ROP) 
goals 


Available to all grantee 
schools (2001-2002 only) 


Executive coaching and 
BASRC staff coaching of 
district leaders. BASRC staff 
coaching of all focal schools 
within focal districts 3 


Meetings 


Summer institutes 

Network meetings by topic 
(for example, data, 
assessment, literacy) 


Summer institutes 

Network meetings by roles 
(for example, district adminis- 
trators, principals, and 
coaches) 


Summer institutes 

Focal district network 
meetings 

Network meetings by roles 


Accountability 


Portfolio review 

Review of progress 
(annual review meetings) 


Review of progress 


Quarterly reviews 
Review of progress 



NOTE: “School-level coaching by BASRC staff was scaled back or discontinued early in the process of implement- 
ing the BASRC focal strategy. As such, most of the coaching in the focal strategy occurred at the district level. 
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In selecting the focal districts, BASRC tended to look to districts where there was al- 
ready a strong working relationship from Phase I and/or Phase II efforts. (As a result, four of the 
six focal districts were part of earlier BASRC efforts.) Districts with achievement gaps and with 
a reform-minded superintendent were also high priorities. The focal schools were then selected 
through negotiations between BASRC and district administrators. 

Like the original Phase II reform model, the BASRC focal strategy emphasized system- 
level reform and was built on three primary design features: 

1 . coaching 

2. evidence-based decision-making 

3. professional networks and collaboration 

However, by 2002-2003 (Year 2 of Phase II and the first year of the focal strategy), 
coaching by BASRC staff or individuals who were selected and trained by BASRC was unique 
to the focal strategy, above and beyond BASRC’ s broader Phase II reform efforts. Coaching 
directly by BASRC staff was dedicated predominantly to focal districts and focal schools. 
Coaching by district or school staff, evidence-based decision-making, and networks are each 
elements of BASRC’ s original Phase II reform model. However, in focal districts and schools 
these efforts were supposed to be reinforced by interactions with BASRC staff. All of these fea- 
tures will be discussed in more detail in Chapter 2. 



The Evaluation of the BASRC Focal Strategy 

The independent evaluation of BASRC is a collaborative effort of MDRC and the Stan- 
ford University Center for Research on the Context of Teaching (CRC). 11 MDRC’s study sheds 
light on the relationship between the BASRC focal strategy and improvements in student 
achievement. This report presents the degree of academic progress in BASRC focal districts, and 
whether or not it outpaces progress in similar districts that were not participating in the BASRC 
focal strategy. BASRC focal reforms focused on literacy instruction and were concentrated at the 
elementary level. As such, all analyses are focused at the elementary-school level (in particular, 
the fifth grade) and emphasize district performance on measures of student literacy. 



n CRC also conducted an evaluation of BASRC’ s first phase of reform work (1995-2001). (See Center for 
Research on the Context of Teaching, 2002.) 
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MDRC’s study does not isolate the effect of the BASRC focal strategy on student 
achievement. Because of the nature of the focal strategy, and the nature of districts’ participa- 
tion in the intervention, it is not feasible to render a single, reliable, unbiased “net impact” esti- 
mate summarizing BASRC’ s effect on student learning. Assessing the net impact of any educa- 
tional reform requires a reliable estimate of outcome levels that would have occurred in the ab- 
sence of the reform. As will be explained in more detail in Chapter 4, because focal districts 
were not selected at random, this is simply not possible in the case of the BASRC focal strategy. 
Nevertheless, MDRC’s study reflects on the relationship between the BASRC focal strategy 
and improvements in student achievement. The study also presents an analysis of the implemen- 
tation challenges associated with the focal strategy and the difficulties BASRC faced as an ex- 
ternal reform organization adjusting to the changing district needs and context and maintaining 
a focus on both “top-down” and “bottom-up” reforms. 

This study informs and is informed by the CRC studies. The CRC studies focus in more 
detail on the process of reform and the relationship between BASRC reform efforts and changes 
at the district, school, and classroom levels. CRC’ s studies are generating detailed information 
about the implementation of particular design features, the school and district contexts in which 
reforms take place, and how both the reform efforts and the reform contexts evolve over time. 
Moreover, the CRC studies are providing important information for the interpretation of the 
findings of the MDRC study (for example, descriptions of the factors driving the changes in 
particular reform practices or of the dynamics limiting or enhancing the connections between 
particular reform practices and improved student achievement). Together, the MDRC and CRC 
studies yield an evaluation that contributes to the knowledge base about local system reform by 
exploring both the process of system reform and the relationship between particular reform 
practices and changes in student achievement. 



Overview of This Report 

The remainder of this report is organized as follows: Chapter 2 explains the theory of 
action behind the BASRC focal strategy. Chapter 3 presents characteristics of BASRC focal 
districts before they started participating in the focal strategy and provides a summary of im- 
plementation issues. Chapter 4 documents the relationship between the BASRC focal district 
strategy and trends in student achievement, focusing on fifth-grade students. Finally, Chapter 5 
discusses hypotheses for explaining the findings. 
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Chapter 2 

The BASRC Focal Strategy 



Theory of Action 

Since the inception of its focal strategy in fall 2002, the Bay Area School Reform Col- 
laborative (BASRC) has continuously refined the theory of action underlying the focal strategy 
in order to be responsive to changing local needs and priorities. However, three key features 
have remained central: 

• coaching 

• evidence-based decision-making 

• professional networks and collaboration 

This section describes these core features, the way they were originally intended to be 
put into practice in focal districts and schools, and the underlying theory linking them to im- 
proved achievement and accelerated learning among the lowest-performing students. This sec- 
tion also includes a description of BASRC’ s “blueprint” for the focal strategy and what might 
be observed if the strategy were implemented as planned. 1 Chapter 3 offers observations based 
on actual implementation of the focal strategy. 

As mentioned earlier, it is important to note that the focal strategy is very similar to the 
Phase II strategy in terms of the key components. What sets the theory for the focal strategy 
apart from the theory for the Phase II strategy is that BASRC provides districts and schools with 
more intensive support for reform, primarily in the form of coaching by BASRC staff and by 
experienced coaches hired by BASRC to support district reform. To support the work of the 
coaches, BASRC also conducts comprehensive needs assessments in the focal districts and 
quarterly meetings to review progress. Coaching (from within the school system), evidence- 
based decision-making, and networks, established as the key components for reform in Phase II, 
are also the key components of the focal strategy. 



'The ongoing changes in BASRC’s model and implementation make such a description challenging, but 
this evaluation focuses on the basic features that appear to remain stable in BASRC’s design over time. 
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Coaching 

BASRC has based its underlying theory of coaching on the idea that educators “need help 
finding, prioritizing, and implementing the many good ideas available [in the field ].” 2 Coaching 
was intended to be the primary delivery system for BASRC focal reforms. BASRC’ s original fo- 
cal strategy included coaching at both the district and school levels. At the outset of their participa- 
tion in the focal strategy, both districts and schools set measurable goals for raising student 
achievement and narrowing the performance gaps among different groups of students. BASRC 
coaches were supposed to work with leaders at both levels to keep them focused on these goals 
and to help them develop and implement strategies for achieving the goals. 

The BASRC theory is that, by providing “practical, useful, and effective support,” its 
coaching will lead to better teaching and improved student achievement . 3 According to 
BASRC, by “diagnosing problems, identifying organizational areas of need, setting goals and 
agreeing on a focus, establishing an instructional plan based on research and best practices in 
high-performing schools and districts, and putting in place a rigorous and thoughtful continuous 
improvement process,” coaching can lead to improved teaching and learning . 4 The coaching 
model does not, however, call for any specific activities designed to ensure particular changes in 
teaching practice, nor does it stipulate how these changes are to occur. 

At the district level, BASRC h ir ed executive coaches with district leadership experience 
to work with superintendents and/or other central office administrators. Their coaching was in- 
tended to address the district’s capacity with respect to five core elements defined by BASRC : 5 

• Leadership: The superintendent and other district leaders should provide and 
articulate a vision for a districtwide focus. They should support that vision by 
allocating resources, providing schools with flexibility, and making sure all 
school staff members have the instructional materials, technology, and tools 
they need to achieve agreed-upon organizational goals. 

• Culture: Teachers and administrators throughout the district should believe that 
every student in the district, regardless of gender, race, primary language, or so- 
cioeconomic status, is capable of meeting district standards when provided with 
effective instruction. Teachers and administrators should strive for and reward 



2 BASRC Web site (2003). 

3 Springboard Schools Web site (2005). 

4 BASRC Web site (2003). 

5 These terms and their definitions summarize the BASRC document entitled “District Critical Ele- 
ments” (2002). 
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excellence, make the best use of time during the school day, ensure safe and 
clean environments, and involve the community in the life of the school. 

• Infrastructure: The board of tmstees and district leaders should adopt reform 
policies that work, properly distribute authority and resources, and regularly 
evaluate progress toward goals. These leaders should also build strong profes- 
sional learning communities, establish effective two-way communications sys- 
tems, and provide well-structured professional development programs. 

• Educational Program: District leaders should define content standards for 
what students should know and performance standards for how students 
should demonstrate their achievement. They should also adopt or develop a 
comprehensive K-12 curriculum aligned with state and national standards 
and reinforced by evidence-based practice, define standards for effective im- 
plementation of programs and practices, and regularly examine data to de- 
termine program effectiveness. 

• Professional Development: District leaders should design professional de- 
velopment that is aligned with the district focus and targeted to improving in- 
struction and accelerating the achievement of low-performing students. 

These professional development opportunities should be regularly evaluated 
and adjusted to better meet the needs of teachers, and they should focus on 
helping teachers and administrators understand the implementation of new 
programs and strategies and use data to inform instruction. 

Executive coaches plan their activities by first leading a needs assessment of the dis- 
trict’ s capacity in each of these areas, and then helping district leaders formulate goals based on 
the results. An important aim of the executive coaching model is to help district leaders become 
reflective and outcome-oriented. Executive coaches, along with BASRC staff, also participated 
in quarterly meetings to discuss district reform progress. This model assumes that the district’s 
organizational culture and the behavior of its leaders can be leveraged in order to put effective 
reform practices into place in schools. BASRC does not, however, prescribe the specific path- 
ways or mechanisms that will translate these behaviors into actual school-level changes in 
teaching and learning. 

Executive coaches were h ir ed to work with superintendents and other leaders at the dis- 
trict level. At the school level, the original design was that BASRC staff would deliver coaching 
to support schools’ efforts to achieve their reform goals. In collaboration with the executive 
coaches, BASRC school coaches were to address any number of school organizational issues 
related to district initiatives, from leadership to performance monitoring. For example, if a dis- 
trict embarks on an initiative to implement a new assessment program, BASRC’ s school-level 
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coaching might focus on helping a school use that assessment data in an analysis of student 
achievement. BASRC school-level coaches work with principals, school- and district-based 
coaches, grade-level teams, and/or other leadership teams. (As Chapter 3 discusses, because of 
implementation challenges, school-level coaching faded from the focal model in many schools 
relatively early in the implementation process.) 

BASRC coaches provide focal districts and schools with specific tools and supports de- 
veloped by BASRC. An example of one such tool is a database developed by Just for the Kids - 
California (JFTK-CA). The database provides districts and schools with state standards assess- 
ment results by grade and includes average results for all students as well as for students who 
have been in the school for at least three years. For any school in the system, JFTK-CA also 
provides the names of the top 10 performing schools in California with comparable student 
demographics. The vision for this system is that a district’ s leaders and teachers will communi- 
cate with these highest-performing schools in order to learn about practices that drive their suc- 
cess. Another example of a tool coaches may use is a worksheet that walks leaders and teachers 
through a Cycle of Inquiry process. The worksheet includes questions for educators to answer, 
allowing them to identify a problem, diagnose it, and develop a practice to address it. 

Coaching by the BASRC staff is a delivery system for the BASRC focal reforms. At 
the same time, coaching by staff within the school system is also a key feature of the BASRC 
focal reforms and of the earlier phases of BASRC. 

Three types of coaching roles were originally specified as part of the broader Phase II 
strategy. These remain as part of the focal strategy as well. First, Local Collaborative coaches 
grew out of the focus on groups of schools working together on reform issues (Local Collabora- 
tives) in Phase II. These coaches, based either in the district office or at a school site, bridge 
BASRC schools with one another and the district office, promoting both bottom-up and top- 
down reform work. They may be district administrators, principals, school-based coaches, or 
individuals hired by the district specifically for the role. Second, reform coordinators are based 
on-site at BASRC schools and are often teachers or other school staff who are released from 
their regular duties on a part-time basis to help advance BASRC reforms. Third, literacy 
coaches are typically part of a district strategy not necessarily related to BASRC. Literacy 
coaches help with the adoption of new curriculum or provide intervention work for struggling 
students. Table 2.1 lists the various coaching roles, their organizational affiliations, and their 
relationship to the different phases of BASRC. 



Evidence-Based Decision-Making 

Evidence-based decision-making is the second key design feature of the BASRC focal 
strategy. BASRC promotes this component through coaching as well as through a variety of 
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using BASRC grants) 





tools and processes. Instead of prescribing a set of specific classroom practices, this component 
articulates a general process for reform of instructional practice. One of the key tools in this 
process is the Cycle of Inquiry, which is a method of identifying, understanding, and meeting 
the needs of struggling students. Educators begin this process by examining student achieve- 
ment data, including diagnostic assessments, to identify problems and determine which students 
are struggling in which areas. BASRC also encourages schools to adopt and give more frequent 
assessments. At the school level, this cycle is designed to enable teachers to better diagnose stu- 
dents’ needs and fine-tune classroom strategies before students move on to the next grade. 
BASRC encourages districts and schools to implement the Cycle of Inquiry at all levels in the 
system (school, grade, and classroom) to identify achievement gaps, and to focus on a specific 
academic area across all levels (typically, literacy). The inquiry model assumes that districts, 
schools, and teachers will work to identify teaching practices that will boost student achieve- 
ment, particularly among the lowest-performing students. 

With the goal of building districts’ and schools’ capacity to engage in evidence-based 
decision-making, BASRC uses a variety of other tools, including annual Reviews of Progress 
(in which districts and schools present evidence about progress in improving performance and 
closing achievement gaps) and the JFTK-CA database. In addition, BASRC encourages districts 
to develop systems for providing schools and teachers with useful and accessible data, and for 
linking the data to systemwide assessment programs. 



Professional Networks and Collaboration 

Professional networks and collaboration make up the third primary design feature of the 
BASRC focal strategy. This feature consists of a variety of opportunities for school leaders to 
convene and share knowledge with each other. For example, BASRC organizes professional de- 
velopment services in four different professional networks for focal district and school leaders: 

1 . the Principals’ Network for principals in all BASRC schools 

2. the Local Collaborative Coaches’ Network for school- and district-based 
coaches in all BASRC districts 

3. the District Leaders' Network for district leaders in all BASRC districts 

4. the Focal Superintendents’ Network for the superintendents in the focal dis- 
tricts only 

BASRC’ s goal for these networks is to build educators’ capacity to address student and 
school performance. Examples of topics include leadership, the Cycle of Inquiry, and equity. 
Other collaboration opportunities are BASRC-organized Summer Institutes for school and dis- 
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trict leadership teams, as well as meetings organized around particular themes, such as literacy 
and the use of data. In all of these meetings, BASRC encourages participants to continue shar- 
ing knowledge and to develop school-level networks at their sites, often providing them with 
tools to pass on what they have learned. 

In addition to the collaboration opportunities BASRC provides at network meetings, the 
organization encourages districts and schools to create collaboration opportunities at their dis- 
trict offices and school sites. BASRC advocates for collaboration that supports what the organi- 
zation refers to as “bottom-up and top-down” reform, in which both schools and districts share 
reform ideas and influence one another. BASRC also promotes collaboration across and within 
schools. The theory is that such collaboration builds knowledge and creates mutual accountabil- 
ity, leading to improved practice. 



Summary 

As illustrated in Figure 2.1, the theory of action underlying the BASRC focal strategy 
suggests that the implementation of the three key design features would yield changes in dis- 
trict, school, and classroom practices hypothesized to improve student outcomes. Taken to- 
gether, coaching, evidence-based decision-making, and networks and collaboration are thought 
to lead to districts and schools becoming more engaged in a continuous improvement process 
that is reflective, evidence-based, and collaborative. This process is supposed to help educators 
find effective ways to improve instruction for all students, and for low-performing students in 
particular, leading to improved and more equitable student outcomes. Because the focal reforms 
are process-oriented, they can result in different outcomes within each district, school, and 
classroom. The outcomes may be or may not be policies or practices that are directly linked 
with improved teaching and learning, or they may be incremental improvements in culture that 
eventually foster better teaching and learning. 
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Figure 2.1 

BASRC Focal Strategy Theory of Action 




District Reform 

Data-based decisions 
High district standards 
Evidence-based practices 
Quality professional development 
Distributed leadership 



Expected Intermediate Outcomes 



School Reform 

Data-based decisions 
Learning communities 
Knowledge sharing 
Distributed leadership 
Focus on achievement gaps 



Student Outcomes 



Improved Instruction 



Data-based decisions 
Focus on achievement gaps 
Diagnostic assessment 
Standard-based instruction 
High peformance standards 




SOURCE: Adapted from Center for Research on the Context of Teaching (CRC), Stanford University, 2004. 







Chapter 3 

Implementation of the BASRC Focal Strategy 



Chapter 2 described the theory of action behind the Bay Area School Reform Collabo- 
rative (BASRC) focal strategy and how the strategy was intended to unfold if put into practice 
as planned. This chapter describes the places in which the focal strategy was implemented 
and summarizes key implementation challenges. 

Overall, the evidence presented in this chapter suggests that: 

• BASRC implemented the focal strategy in a set of districts that were similar 
to, though slightly more disadvantaged than, the Bay Area as a whole. 

• At the district level, the focal strategy was characterized by coaching by an ex- 
ecutive coach, some additional coaching or support from other BASRC staff, a 
separate BASRC-led Focal Superintendents’ Network, and review meetings 
with BASRC staff. These activities led to goals for improving student 
achievement and reducing achievement gaps. However, they did not typically 
translate into specific reforms aimed at improving teaching and learning. 

• Although the BASRC focal strategy originally included a model aimed at a set 
of “focal schools” within the focal districts, this school-level component was 
not implemented as planned. As a result, there was no meaningful distinction 
between focal and non-focal schools in program resources or implementation. 

• There was evidence of all three key features of the focal strategy in schools 
in the BASRC focal districts. However, these were likely vestiges of earlier 
BASRC reform efforts. Also, the extent to which teachers engaged in these 
BASRC practices did not increase during the years of the focal strategy. 

The findings described in this chapter add to general knowledge about implementing 
districtwide reforms and about the role external support organizations play in supporting dis- 
trictwide reforms, particularly those that emphasize coaching as a primary means of support. 
The findings illustrate how hard it can be for external organizations to develop lasting coaching 
relationships, to effectively influence processes and culture, to simultaneously prioritize both 
district- and school-level reform efforts, and to effectively translate their efforts into reforms 
that improve teaching and learning. 
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The BASRC Focal Districts 

This report focuses on five of the six BASRC focal districts. 1 All five districts are in the 
San Francisco Bay Area. As mentioned earlier, BASRC selected the focal districts, looking in 
particular to districts where there was already a strong working relationship from Phase I and/or 
Phase II efforts. As a result, four of the six focal districts were part of earlier BASRC efforts. 
Achievement gaps and a reform-minded superintendent were also important criteria in the se- 
lection of focal districts. Within each focal district, two to five focal schools were selected 
through negotiations between BASRC and each superintendent. The schools selected may have 
been elementary, middle, or high schools. As explained earlier, MDRC’s research efforts in re- 
gard to BASRC have focused solely on elementary schools. There were a total of 12 focal 
schools at the elementary level. Table 3.1 lists the five focal districts and the elementary -level 
focal schools within them. 

Looking at the five BASRC focal districts in this evaluation, Table 3.2 compares the 
districts’ fifth-grade demographic characteristics and achievement levels before implementation 
of the BASRC focal strategy with those of other districts in the San Francisco Bay Area and 
throughout California. Achievement levels are measured by performance on both the Stanford 
Achievement Test, Ninth Edition (SAT-9), a nationally norm-referenced test, and the California 
Standards Tests (CSTs), which measure student performance relative to California state stan- 
dards. The baseline, or preimplementation, period for the BASRC focal strategy for which 
MDRC has data includes the 1999-2000, 2000-2001, and 2001-2002 school years. 3 The base- 
line demographic characteristics and student achievement levels for fifth-grade students, shown 
in Table 3.2, represent averages over these three years. Note, however, that CST scores for read- 
ing were available only for 2000-2001 and 2001-2002 (except for the below- or far-below-basic 
average, which is based on one baseline year, 2001-2002) and, for math, only for 2001-2002. 

Table 3.2 shows that, relative to Bay Area districts, 4 BASRC focal districts served a 
smaller proportion of white students and a slightly larger proportion of Hispanic and Asian stu- 
dents, but similar proportions of black students. 5 For example, on average, 43 percent of the 



'Because BASRC’s sixth focal district. Oak Grove, joined the initiative one year after the start of the focal 
strategy and is following an action plan different from that of the original five districts, MDRC is not including 
it in the evaluation sample. 

“Bay Area districts are defined as those districts in Alameda, Contra Costa, Marin, San Francisco, San Mateo, 
and Santa Clara counties. All districts and schools with students in the fifth grade were included in the samples. 

3 Data for these analyses are from the California Department of Education. For more information on the 
tests, see Appendix A. 

4 This includes focal districts and districts that participated in earlier phases of BASRC reforms. 

5 The “Other” category includes Native American students and students of multiple ethnicities. The 
“Asian” category includes Pacific Islander and Filipino students. 
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Table 3.1 



BASRC Focal Districts and Their Focal Schools 



Focal District 


Focal Schools 


Newark Unified School District 


Snow Elementary 
Musick Elementary 
Milani Elementary 


Alameda Unified School District 


Paden Elementary 
Woodstock Elementary 


Pacifica School District 


Linda Mar Elementary 
Vallemar Elementary 


San Rafael City Elementary and High 
School Districts 


Bahia Vista Elementary 
San Pedro Elementary 
Laurel Dell Elementary 


San Bruno Park Elementary 


El Crystal Elementary 


School District 


John Muir Elementary 



SOURCE: MDRC field research data. 

NOTE: Focal districts are districts participating in the BASRC focal strategy. Focal schools are 
schools participating in the BASRC focal strategy. 



fifth-grade students served by BASRC focal districts were white, compared with 53 percent in 
the Bay Area districts. At the same time, 26 percent of the fifth-graders in BASRC focal dis- 
tricts were Hispanic, compared with 22 percent of those in the other Bay Area districts. Table 
3.2 also shows that BASRC focal districts had slightly higher percentages of English Language 
Learners and students eligible for a free or reduced-price lunch (a common proxy for economic 
disadvantage). During the baseline period, an average of 16 percent of fifth-grade students in 
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Table 3.2 



Characteristics of BASRC Focal, Bay Area, and California Districts, Fifth-Grade 
Averages for School Years 1999-2000 Through 2001-2002 



Characteristic 


BASRC Focal 
Districts 


All Bay Area 
Districts 2 


All California 
Districts 


Demosrauhic characteristics 

Race/Ethnicity (%) 


White 


43.1 


52.7 


55.4 


Hispanic 


26.2 


21.6 


30.6 


Asian 


21.9 


16.3 


5.7 


Black 


6.2 


7.6 


3.9 


Other 


2.5 


1.9 


3.8 


English language learners (%) 


15.5 


14.0 


17.7 


Eligible for free or reduced priced lunch b (%) 


29.8 


22.4 


44.6 


Achievement levels 

California Standards Tests (CSTs) 
Language arts c 


Proficient or advanced (%) 


37.8 


47.0 


33.0 


Basic (%) 


39.8 


33.4 


38.9 


Below or far below basic (%) 


22.6 


19.6 


28.0 


Mean scale score 


338.4 


348.4 


332.0 


Math d 


Proficient or advanced (%) 


37.2 


44.0 


31.2 


Basic (%) 


32.6 


27.6 


31.4 


Below or far below basic (%) 


30.4 


28.3 


37.4 


Mean scale score 


335.6 


347.2 


325.5 


Stanford-9 (SAT-9) tests 
Reading 


At or above 50th National Percentile Rank (%) 


56.0 


63.9 


51.2 


25th-49th National Percentile Rank (%) 


23.4 


18.3 


22.6 


Below 25th National Percentile Rank (%) 


20.6 


17.7 


26.2 


Mean scale score 


659.1 


667.2 


654.2 


Math 


At or above 50th National Percentile Rank (%) 


63.5 


68.0 


58.0 


25th-49th National Percentile Rank (%) 


17.7 


14.7 


18.6 


Below 25th National Percentile Rank (%) 


18.7 


17.3 


23.5 


Mean scale score 


659.6 


666.9 


654.7 


Sample size 


5 


106 


944 



(continued) 
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Table 3.2 (continued) 



SOURCE: Publicly available district- and school-level data files from the California Department of Education. 

NOTES: Due to rounding, percentages may not sum to 100 percent. The estimates in the table represent averages 
across all districts, regardless of district size. 

a Bay Area districts include all districts with students in grade 5 (including BASRC focal districts) in Alameda, 
Contra Costa, Marin, San Francisco, San Mateo, and Santa Clara counties. 

b Averages for free or reduced-price lunch are presented at the school level only. 

L Due to availability of test scores, California Standards Tests averages in language arts are based on just two 
baseline years, 2000-2001 and 2001-2002, except for the "Below or far below basic" average, which is based on 
one baseline year, 2001-2002. 

d Due to availability of test scores, California Standards Tests averages in math are based on just one baseline 
year, 2001-2002. 



the focal districts were English Language Learners, compared with 14 percent of students in 
Bay Area districts as a whole. Also, 30 percent of students in focal districts were eligible for the 
free or reduced-price lunch program, compared with 22 percent in the rest of the Bay Area. 6 

This table also compares characteristics of fifth-grade students in the BASRC focal dis- 
tricts with students in all districts in the State of California. Contrary to the comparison with 
Bay Area districts, the BASRC focal districts included a smaller proportion of Hispanic students 
and more Asian students relative to the whole state. BASRC focal districts also served a similar 
proportion of students who were English Language Learners. Lastly, both the focal districts and 
the Bay Area districts served much lower proportions of students eligible for free or reduced- 
price lunch than the state average. 

Students in the focal districts generally demonstrated lower levels of reading achievement 
and math achievement on the CST compared with students in the Bay Area as a whole. This is 
consistent with BASRC’ s efforts to target its interventions toward relatively low-performing dis- 
tricts, particularly with respect to literacy. However, the extent to which the achievement levels 
differ varies by subject. In particular, differences are larger for reading scores than they are for 
math. The patterns of test scores on the SAT-9, also presented in Table 3.2, seem to show similar 
trends. In contrast, compared with all districts in the state of California, students in the BASRC 
focal districts have higher levels of achievement in both reading and math. 

In sum, while not all the differences are large, fifth-grade achievement levels in 
BASRC focal districts are lower than in other Bay Area districts for both the CST and SAT-9. 



6 Note that the figures for eligibility for a free or reduced-price lunch represent school-level percentages 
rather than grade-level percentages. 
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